Two Steps at a Time---Taking GAN Training in Stride with Tseng's Method

Motivated by the training of Generative Adversarial Networks (GANs), we study methods for solving minimax problems with additional nonsmooth regularizers. We do so employing \emph{monotone operator} theory, in particular \emph{Forward-Backward-Forward (FBF)} method, which avoids known issue limit cycling correcting each update a second gradient evaluation. Furthermore, propose seemingly new scheme recycles old gradients to mitigate computational cost. In doing rediscover related \emph{Optimistic Gradient Descent Ascent (OGDA)}. For both schemes prove novel convergence rates convex-concave via unifying approach. The derived error bounds are terms gap function ergodic iterates. deterministic and stochastic problem show rate O(\nicefrac1k) O(\nicefrac1k), respectively. complement our theoretical results empirical improvements Wasserstein GANs on CIFAR10 dataset.